Skip to content

✨ Performance Alerting #2081

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

✨ Performance Alerting #2081

wants to merge 1 commit into from
+116 −5

Conversation

dtfranz
Copy link
Contributor

@dtfranz dtfranz commented Jul 8, 2025 •
edited
Loading

Description

Introduces an early-warning series of prometheus alerts to attempt to catch issues with performance at an early stage in development.

As the e2e tests run, the installed prometheus instance is scraping metrics from catalogd and operator-controller, and will fire alerts based on rules introduced in this PR. Since we're running these tests on the github runners which do not have consistent performance, our alerts must be based on platform-independent metrics and are therefore limited. Any other ideas for metrics to check on this PR are appreciated!

Once the e2e tests finish, prometheus is queried for active alerts. Any alerts found in pending state will result in a warning being set on the e2e workflow. Any alerts in firing state will give an error. These errors do not (at the moment) fail the run, but are visible when the workflow details are viewed.

For instance:

Prometheus Alert Pending
operator-controller-memory-growth: operator-controller pod memory usage growing at a high rate for 5 minutes: 72.86kB/sec

I am still in the process of tuning the alerts, so at the moment I am not making this a required check.

Potential Enhancements:

  • Additional alerts, if any
  • Fine-tune the alerts and fail runs when they fire
  • Remove yaml from script and organize into an additional kustomization component
  • Output metrics as a mermaid XY plot in the workflow summary

Closes #1904
Closes #1905

Reviewer Checklist

  • API Go Documentation
  • Tests: Unit Tests (and E2E Tests, if appropriate)
  • Comprehensive Commit Messages
  • Links to related GitHub Issue(s)

@dtfranz dtfranz requested a review from a team as a code owner July 8, 2025 14:50
@openshift-ci openshift-ci bot requested review from perdasilva and trgeiger July 8, 2025 14:50
@openshift-ci OpenShift CI
Copy link

openshift-ci bot commented Jul 8, 2025

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign joelanford for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@dtfranz dtfranz force-pushed the metrics-alerting branch 2 times, most recently from 97a268f to cb81424 Compare July 8, 2025 14:53
@netlify Netlify
Copy link

netlify bot commented Jul 8, 2025 •
edited
Loading

✅ Deploy Preview for olmv1 ready!

Name Link
🔨 Latest commit bb8a597
🔍 Latest deploy log https://app.netlify.com/projects/olmv1/deploys/686dd6d1f619a80008027da5
😎 Deploy Preview https://deploy-preview-2081--olmv1.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

tmshort
tmshort reviewed Jul 8, 2025
View reviewed changes
@dtfranz
Performance Alerting
bb8a597 
Introduces an early-warning series of prometheus alerts to attempt to catch issues with performance at an early stage in development.

Signed-off-by: Daniel Franz <dfranz@redhat.com>
@dtfranz dtfranz force-pushed the metrics-alerting branch from cb81424 to bb8a597 Compare July 9, 2025 02:41
@codecov Codecov
Copy link

codecov bot commented Jul 9, 2025 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 73.41%. Comparing base (1333f7b) to head (bb8a597).
Report is 4 commits behind head on main.

Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #2081      +/-   ##
==========================================
+ Coverage   73.35%   73.41%   +0.06%     
==========================================
  Files          77       77              
  Lines        7056     7076      +20     
==========================================
+ Hits         5176     5195      +19     
- Misses       1540     1541       +1     
  Partials      340      340
Flag Coverage Δ
e2e 44.90% <ø> (-0.08%) ⬇️
experimental-e2e 51.20% <ø> (-0.15%) ⬇️
unit 58.38% <ø> (+0.10%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tmshort tmshort tmshort left review comments

@perdasilva perdasilva Awaiting requested review from perdasilva

@trgeiger trgeiger Awaiting requested review from trgeiger

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Select detection criteria for CI failure of e2e metrics job Create/Modify upstream CI job
2 participants
@dtfranz @tmshort